20 research outputs found
Implicit Langevin Algorithms for Sampling From Log-concave Densities
For sampling from a log-concave density, we study implicit integrators
resulting from -method discretization of the overdamped Langevin
diffusion stochastic differential equation. Theoretical and algorithmic
properties of the resulting sampling methods for and a
range of step sizes are established. Our results generalize and extend prior
works in several directions. In particular, for , we prove
geometric ergodicity and stability of the resulting methods for all step sizes.
We show that obtaining subsequent samples amounts to solving a strongly-convex
optimization problem, which is readily achievable using one of numerous
existing methods. Numerical examples supporting our theoretical analysis are
also presented
Newton-Type Methods for Non-Convex Optimization Under Inexact Hessian Information
We consider variants of trust-region and cubic regularization methods for
non-convex optimization, in which the Hessian matrix is approximated. Under
mild conditions on the inexact Hessian, and using approximate solution of the
corresponding sub-problems, we provide iteration complexity to achieve -approximate second-order optimality which have shown to be tight.
Our Hessian approximation conditions constitute a major relaxation over the
existing ones in the literature. Consequently, we are able to show that such
mild conditions allow for the construction of the approximate Hessian through
various random sampling methods. In this light, we consider the canonical
problem of finite-sum minimization, provide appropriate uniform and non-uniform
sub-sampling strategies to construct such Hessian approximations, and obtain
optimal iteration complexity for the corresponding sub-sampled trust-region and
cubic regularization methods.Comment: 32 page
Convergence of Newton-MR under Inexact Hessian Information
Recently, there has been a surge of interest in designing variants of the
classical Newton-CG in which the Hessian of a (strongly) convex function is
replaced by suitable approximations. This is mainly motivated by large-scale
finite-sum minimization problems that arise in many machine learning
applications. Going beyond convexity, inexact Hessian information has also been
recently considered in the context of algorithms such as trust-region or
(adaptive) cubic regularization for general non-convex problems. Here, we do
that for Newton-MR, which extends the application range of the classical
Newton-CG beyond convexity to invex problems. Unlike the convergence analysis
of Newton-CG, which relies on spectrum preserving Hessian approximations in the
sense of L\"{o}wner partial order, our work here draws from matrix perturbation
theory to estimate the distance between the subspaces underlying the exact and
approximate Hessian matrices. Numerical experiments demonstrate a great degree
of resilience to such Hessian approximations, amounting to a highly efficient
algorithm in large-scale problems.Comment: 32 pages, 10 figure
A Newton-MR algorithm with complexity guarantees for nonconvex smooth unconstrained optimization
In this paper, we consider variants of Newton-MR algorithm for solving
unconstrained, smooth, but non-convex optimization problems. Unlike the
overwhelming majority of Newton-type methods, which rely on conjugate gradient
algorithm as the primary workhorse for their respective sub-problems, Newton-MR
employs minimum residual (MINRES) method. Recently, it has been established
that MINRES has inherent ability to detect non-positive curvature directions as
soon as they arise and certain useful monotonicity properties will be satisfied
before such detection. We leverage these recent results and show that our
algorithms come with desirable properties including competitive first and
second-order worst-case complexities. Numerical examples demonstrate the
performance of our proposed algorithms
Newton-MR: Inexact Newton Method With Minimum Residual Sub-problem Solver
We consider a variant of inexact Newton Method, called Newton-MR, in which
the least-squares sub-problems are solved approximately using Minimum Residual
method. By construction, Newton-MR can be readily applied for unconstrained
optimization of a class of non-convex problems known as invex, which subsumes
convexity as a sub-class. For invex optimization, instead of the classical
Lipschitz continuity assumptions on gradient and Hessian, Newton-MR's global
convergence can be guaranteed under a weaker notion of joint regularity of
Hessian and gradient. We also obtain Newton-MR's problem-independent local
convergence to the set of minima. We show that fast local/global convergence
can be guaranteed under a novel inexactness condition, which, to our knowledge,
is much weaker than the prior related works. Numerical results demonstrate the
performance of Newton-MR as compared with several other Newton-type
alternatives on a few machine learning problems.Comment: 35 page
Non-Uniform Smoothness for Gradient Descent
The analysis of gradient descent-type methods typically relies on the
Lipschitz continuity of the objective gradient. This generally requires an
expensive hyperparameter tuning process to appropriately calibrate a stepsize
for a given problem. In this work we introduce a local first-order smoothness
oracle (LFSO) which generalizes the Lipschitz continuous gradients smoothness
condition and is applicable to any twice-differentiable function. We show that
this oracle can encode all relevant problem information for tuning stepsizes
for a suitably modified gradient descent method and give global and local
convergence results. We also show that LFSOs in this modified first-order
method can yield global linear convergence rates for non-strongly convex
problems with extremely flat minima, and thus improve over the lower bound on
rates achievable by general (accelerated) first-order methods